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1. Introduction 

Let a £ S n and r £ Sk be two permutations. We say that a contains r if 
there exists a subsequence 1 ^ i\ < i% < • • • < i& ^ n such that (a^, . . . , cti k ) is 
order-isomorphic to r; in such a context r is usually called a pattern. We say that 
a avoids t, or is r-avoiding, if such a subsequence does not exist. The set of all 
r-avoiding permutations in S n is denoted S n (r). For an arbitrary finite collection 
of patterns T, we say that a avoids T if a avoids any r £ T; the corresponding 
subset of S n is denoted S n (T) . 

While the case of permutations avoiding a single pattern has attracted much 
attention, the case of multiple pattern avoidance remains less investigated. In par- 
ticular, it is natural, as the next step, to consider permutations avoiding pairs of 
patterns n, r 2 . This problem was solved completely for Tx,T2 £ S3 (see [SS]), for 
n £ S3 and r 2 £ S4 (see [W,A]), and for ti,t 2 £ S4 (see [Bl,Km] and references 
therein). Several recent papers [CW,MVl,Kr,MV2] deal with the case t\ £ S3, 
T2 £ Sk for various pairs t\, T2- Another natural question is to study permuta- 
tions avoiding t\ and containing r 2 exactly t times. Such a problem for certain 
T\,T2 £ S3 and t = 1 was investigated in [R], and for certain t\ £ S3, r 2 £ Sk 
in [RWZ,MVl,Kr]. The tools involved in these papers include continued fractions, 
Chebyshev polynomials, and Dyck paths. 

In this paper we present a general approach to the study of permutations avoiding 
132 and avoiding an arbitrary pattern r £ Sk (or containing it exactly once). As a 
consequence, we derive all the previously known results for this kind of problems, 
as well as many new results. 

The paper is organized as follows. The case of permutations avoiding both 132 
and t is treated in Section 2. We derive a simple recursion for the corresponding 
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generating function for general r. This recursion can be solved explicitly for sev- 
eral interesting cases, including 2-layered and 3-layered patterns (sec [B2, MV2]) 
and wedge patterns denned below. It allows also to write a Maple program that 
calculates the generating function for any given r. This program can be obtained 
from the authors on request. Observe that if r itself contains 132, then any 132- 
avoiding permutation avoids r as well, so in what follows we always assume that 
re 5 fe (132). 

The case of permutations avoiding 132 and containing r exactly once is treated 
in Section 3. Here again we start from a general recursion, and then solve it for 
several particular cases. 

Most of the explicit solutions obtained in Sections 2, 3 involve Chebyshev poly- 
nomials of the second kind. Several identities used for getting these solutions are 
presented in Section 4. The authors are grateful to the referee for explaining to us 
a general approach to such identities. 

The final version of this paper was written during the second author's (A.V.) 
stay at Max-Planck-Institut fur Mathematik in Bonn, Germany. A.V. wants to 
express his gratitude to MPIM for the support. 

2. Avoiding a pattern 

Consider an arbitrary pattern t = (n, . . . , Tfc) € Sfe(132). Recall that Tj is said 
to be a right-to-left maximum if Tj > tj for any j > i. Let mo = ft, mi, . . . , m r be 
the right-to- left maxima of r written from left to right. Then r can be represented 

as 

r = (T°,m ,T 1 ,m ll . . . ,r r ,m r ), 

where each of t % may be possibly empty, and all the entries of t 1 are greater 
than TOj+i and all the entries of r* +1 . This representation is called the canonical 
decomposition of r. Given the canonical decomposition, we define the ith prefix of 
t by it 1 = (r°,m , . . . , r*,TOj) for 1 ^ i < r and 7r° = r°, 7r _1 = 0. Besides, the 
ith suffix of r is defined by o l = (r ! ,m„..., r r , m r ) for < i < r and er r+1 = 0. 
Strictly speaking, prefices and suffices themselves are not patterns, since they are 
not permutations (except for tt t = o° = t). However, any prefix or suffix is order- 
isomorphic to a unique permutation, and in what follows we do not distinguish 
between a prefix (or suffix) and the corresponding permutation. 

Let / r (n) denote the number of permutations in 5„(132) avoiding t, and let 
F T (x) = J2 n ^a fr( n ) xn be the corresponding generating function. By /£(n) we 
denote the number of permutations in 5 n (132) avoiding r and containing p. The 
following proposition is the base of all the other results in this Section. 

Theorem 2.1. For any t £ 5fe(132), F T (x) is a rational function satisfying the 
relation 

r 

F T (x) = l + xJ2(F^ (x) - F^j-i {x))F ai (x). 

3=0 

Proof. Let a e S , „(132,t). Choose t so that a t — n, then a = (a',n,a"), and a 
avoids 1 32 if and only if a' is a permutation of the numbers n — t + l,n—t+2, . . . ,n, 
a" is a permutation of the numbers 1, 2, . . . , n — t, and both a' and a" avoid 132. 
On the other hand, it is easy to see that a contains r if and only if there exists i, 
< i ^ r + 1, such that a' contains 7r l_1 and a" contains o l . Therefore, a avoids 
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t if and only if there exists i, ^ i ^ r, such that a' avoids 7r* and contains n l 1 , 
while a" avoids a 1 . We thus get the following relation: 

n r 
t=l j=0 

To obtain the recursion for F T (x) it remains to observe that 

/;r 1 (o+/^- 1 (o = /^(o 

for any I and j, and to pass to generating functions. Rationality of F T (x) follows 
easily by induction. □ 

Though elementary, the above Theorem enables us to derive easily various known 
and new results for a fixed k. 

Example 2.1 (see [SS, Pr. 11]). Let us find F321 (x). The canonical decomposition 
of 321 is (0, 3, 0, 2, 0, 1), so r — 2, and hence Theorem 2.1 gives 

F 3 2i(x) = 1 + xF 32 {x)F 21 {x) + x(F 32 i(x) - F 32 (x))F 1 {x). 

Since evidently F\(x) = 1 and F 2 i{x) — F 32 (x) = (1 — x) -1 , we finally get 

l-2x + 2x 2 
FMX) = (!-»)» • 



Example 2.2 (see [W]). Let us find i ? 32i4(a;)- The canonical decomposition is 
(321,4), so r = 0, and hence £3214(2) = 1 + X-F321 (2^3214(2;). Thus finally 

/ \ (l-xf 

F ^ X ) ~ l _ 4x + 5x 2_ 3x3 - 



A Maple program that calculates F T (x) for any given r is available from the 
authors on request. 

The case of varying k is more interesting. As an extension of Example 2.1, let us 
consider the case r = (k) = (k, k — 1, . . . , 1). We denote by $(x, y) the generating 
function J2k^i F (k) {x)y k . 



Theorem 2.2. 



_ y(l + x- xy) -yy/(l + x- xy) 2 ^~Ix 
(Xiy) ~ 2x(l-y) 



Proof. Indeed, Theorem 2.1 yields 
fe-i 

F{k) (x) = l + x^2(F u+1) (x) - F {j) {x))F {k _j) (x) + xF {1) {x)F {k _ 1) (x). 
i=i 
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Multiplication by y k and summation over k ^ 2 gives 

<t>{x, y) = + x(l - y)${x, y) | ^'^ - 1 - y) I + y), 



i -y \ y 

and the result follows. □ 

Observe that as a consequence of Theorem 2.2 we get 

1 - J\ — Ax 

lim F (fc) (a;) = lirn(l - y)<f>(x, y) = , 

fc^oo y^l Zx 

in which we recognize the generating function of Catalan numbers. This is a pre- 
dictable result, since as k tends to oo the restriction posed by r vanishes, and we 
end up with just 132-avoiding permutations, which are enumerated by Catalans. 

Let us consider now a richer class of patterns r. We say that r G Sk is a 
layered pattern if it can be represented as r = (r°, r 1 , . . . , r r ), where each of t % is a 
nonempty permutation of the form t 1 = (m i+ i + 1, m i+ i + 2, . . . , m^) with k = m > 
mi > ■ ■ ■ > m r > m r+ i — 0; in this case we denote r by [mo, . . . ,m r ]. Observe 
that our definition slightly differs from the one used in [B2, MV2]: their layered 
patterns are exactly the complements of our layered patterns. It was revealed 
in several recent papers (see [CW, MV1, Kr] and especially [MV2]) that layered 
restrictions are intimately related to Chebyshev polynomials of the second kind 
U p (cos6) = sin(p+ 1)9/ sin 9. Following [MV1], introduce 

Rp(x) = — -, - r-. 

V*u P (^) 

It was proved by different methods in [CW, MV1, Kr] that F[ k ](x) — Rk(x). 
Our next result is an easy consequence of Theorem 2.1. 

Theorem 2.3. Let r ^ 1 and k = m > m\ > • • • > m r , then 

(l — xR mo — nll — \{x) — xR mr (a;)) -F 1 [ mo ,...,m r ] (x) 1 lfimo-mi-l( I )^'[mi r ..,m r ] (x) 
r 

~t~ X ^ ^ ^[mo-m^.-.m^i-mj] {x) (-f 1 [m J _i,...,m r ] {x) ^[ m j ,...,m r ] (^)) • 
J=2 



Proof. Evidently, (f , mo, . . . , f r , m r ) with f* = (m^i + 1, . . . , to* — 1) is the canon- 
ical decomposition of [mo, . . . , m r ]. Next, F a i (x) — F[ m( mr ] (x) for ^ i ^ r — 1, 

and F 7r i(x) = i*[ mo _ m . +1; ... im ._ TO . +l] (a;) for 1 ^ i ^ r. Besides, F„r(x) = F [mr] (x) = 
R mr (x) and F 7T o(x) — Ft mo _ mi _ii(x) = R ma - mi -i{x)- The rest follows from The- 
orem 2.1 via simple algebraic transformations. □ 

For small r one can find explicit expressions for f[ mo .... jrar ](i). 
For r = 1 we get the following generalization of [CW, Th. 3.1, third case] and 
[Kr, Th. 6]. 
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Theorem 2.4. For any k > m > 0, 

F[ k , m ](x) = R k (x). 

Proof. By Theorem 2.3, 

(l - xR k - m -i(x) - xR m (x))F [k ^ m] (x) = 1 - xR k - m -i(x)F [m] (x), 

and the result follows immediately from Lemma 4.1(iv,v) for a — k — m — 1 and 
b — m. □ 

The case r = 2 is more complicated. 

Theorem 2.5. For any k > mi > m 2 > 0, 

t/ a+/3 (z)L r a+7 -i(z)t/ /?+7 (z) + Up-^Upjz) 

where a = k — mi, [3 = mi — ni2, 7 = m2, and z = 1/2^/x. 
Proof. Indeed, by Theorem 2.3, 

(l - xR k - mi -i(x) - xR m2 (x)) F [k ^ mum2] (x) = 1 - a;i? fc _ mi _i(x)i ;l [ mi ^ m2 ](x) + 

x F[k-m 2 ,m 1 -m 2 ]{ x ){F[m 1 ,m 2 ]{ x ) — ^[m 2 ] ( x )) ■ 

Taking into account Theorem 2.4 one gets 
1-xRk — nil — 1 

(x)R mi (x) + xRk-m 2 (x) (R mi (x) - R m2 (x)) 



^[/c,mi ,7712] ( X ) 



1 - xRk- mi -i(x) - xRm 2 (x) 



By Lemma 4.1(iv) for a = k — mi — 1 and b = mi, Lemma 4.1(vi) for a = mi and 
b = mi, and Lemma 4.1 (v) for a = k — mi — 1 and b = m.2, one has 

1 - xR k - mi -i(x)R mi (x^ - - 



a;i?fe-m 2 ( a; )(- R mi(a;) - R m2 (x)) = 



U k - mi -i{z)U mi {zy 

Uk-m 2 -l(z)U mi - m2 -i(z) 
U k -m 2 (z)U mi (z)U m2 (z) 
U k - mi -l(z)U m2 (z) 



1 - xRk- mi -i(x) - xR m2 (x) \/xUk- mi + m2 {z) 
respectively, and we thus get 

U a+ f j (z)U a+f 3 +1 -i{z)U 1 (z) + U a -i(z)U0-i(z)U a+ p-i(z) 



-P]fc,mi,m 2 ] i x ) — 



s/xU a+0 {z)U a+1 {z)Up +1 {z) 



Finally, we use Lemma 4.1 (i) for s = a + 7 — 1, t = [3 + 7, and w = (3 and 
Lemma 4.1(h) for s = f3, t = 0, and w = a — 1 to get the desired result. □ 

For a further generalization of Theorem 2.4, consider the following definition. We 
say that r € S k is a wedge pattern if it can be represented as r = (r , p , . . . , r r , p r ) 
so that each of r* is nonempty, (p 1 , p 2 , . . . , p r ) is a layered permutation of 1, . . . , s 
for some s, and (r 1 , r 2 , . . . , r r ) = (s + 1, s + 2, . . . , k). For example, 645783912 and 
456378129 are wedge patterns. Evidently, [k,m] is a wedge pattern for any m. 
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Theorem 2.6. F T (x) — Rk{x) for any wedge pattern r € 5^(132). 

Proof. We proceed by induction on r. If r = 1 then r = [fc, m] for some m, and 
the result is true by Theorem 2.4. For an arbitrary r > 1, take the canonical 
decomposition of r. Evidently it looks like r = (r',fc,p r ,m), where (p r ,m) = //, 
provided // is nonempty. Therefore, Theorem 2.1 together with F p r (x) — i<] m ] (x) = 
R m {x) give 

_ l-xF T ,{x)R m {x) 
r[ ' l-xF T ,{x)-xR m {xy [ ' 

If r r = (fc), then r' is itself a wedge pattern on fc— to— 1 elements, so by induction 
F T i{x) = Rk-m-i{x), hence the result follows from (2.1) and Lemma 4.1(iv,v) for 
a = k — m — 1 and b = m. Let r r = (I + 1, . . . , fc), then applying Theorem 2.1 
repeatedly fc — I — 1 times we get 



1 



1 



xF T n{x) 



where the height of the fraction equals fc — I — 1 and t" = (r 1 , p 1 , . . . , r r_1 , p r_1 ) is 
a wedge pattern on I — m elements. So, by induction, F T »(x) = Ri- m (x); applying 
Lemma 4.1 (iii) repeatedly fc — I — 1 times we again get F T /(x) = Rk- m -i(x), and 
proceed exactly as in the previous case. The case p r = is treated in a similar 
way. □ 

Remark. A comparison of Theorem 2.6 with the main result of [MV2] suggests that 
there should exist a bijection between the sets 5„(321, [k,m]) and 5„(132, r) for 
any wedge pattern r. However, we failed to produce such a bijection, and finding 
it remains a challenging open question. 

3. Containing a pattern exactly once 

Let g T (n) denote the number of permutations in S , „(132) that contain r e 
5fc(132) exactly once, and g£(n) denote the number of permutations in 5„(132,p) 
that contain r e 5^(132) exactly once. We denote by G T (x) and G^.(x) the corre- 
sponding ordinary generating functions. 

The following statement is similar to Theorem 2.1. 

Theorem 3.1. Let r = (r°, m , . . . , r r , m r ) be the canonical decomposition of r e 
5 fc (132), tften 

r 

(1 - xF^o (x) - xF a r ( X )) G T (x) =xY" GlU (x)G^ 1 (x) 



/or r > 1, and 



G ^ _ xF T (x)G 7r o(x) 



1 - xF^a (x) 

for r = 0. 

Proof. Let a G 5„(132) contain r exactly once, and take the same decomposition 
a = (a', n, a") as in the proof of Theorem 2.1. Similarly to this proof, a contains r 
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exactly once if and only if either a 1 avoids 7r° and a" contains a exactly once, or 
a' contains ir r exactly once and a" avoids a r , or there exists i, 1 < i ^ r, such that 
a' avoids tt 1 and contains ir^ 1 exactly once, while a" avoids o l ~ x and contains a % 
exactly once. We thus get the relation 



9 An) = J2 /*°(* - - f )+J29r(t- l)/„r(n - i) 

t=i t=i 

n r 

+EE^(*- 1 )f:r I ("-i). 
t=i j=i 

and the result follows. □ 

Remark. Strictly speaking, Theorem 3.1, unlike Theorem 2.1, is not a recursion for 
G T {x), since it involves functions of type G P T (unless r = 0). However, for these 
functions one can write further recursions involving similar objects. For example, 

(1 - xF Ti {x) - xF n o(x))GlU(x) = x Y^GS-^Gft- 1 (x), 

i=i 

where cr]_ 1 is the ith suffix of 7T? -1 . Though we have not succeeded to write down 
a complete set of equations in the general case, it is possible to do this in certain 
particular cases. 

Example 3.1 (see [MV1, Th. 3.1]). Let r = [k] = (1, 2, . . . , k). Then r = 0, and 
it follows from Theorem 3.1 that 

gF [fc] (a;)G [fc -i](a;) 

Since i*[fc](a;) = Rk(x) and i?fc(x)(l — a;i?fc_i (x)) = 1 (see (2.2)), we get G[k](x) — 
xR^(x)G[k-i](x) , which together with G^(x) = 1 gives 

i 

G [k ]{x) 



(2^) 



Similarly to Section 2, we consider now the case r = (k) = (k, k — 1, . . . , 1) and 
denote by ^(x, y) the generating function X)fe>i ^<fc) ( x )y k - The following statement 
is a counterpart of Theorem 2.2 

Theorem 3.2. 



_ (1 - aQ(l - sy) - y/(l - x) 2 (l - xyf - 4x 2 (l - x)y 
Proof. Indeed, Theorem 3.1 yields 



fc-i 



(1 - x)G {k) (x) = xG$ (x)G\H 1) (x)+xJ2 (x). 



3=2 
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Observe that if a contains (j + 1) then it contains at least j + 1 copies of (j), 
hence G^ 1 ^ (x) = G^(x) for any j ^ 1. Besides, G^(x) = 1 and G^(x) = x. 
Therefore, multiplication of the above equation by y k and summation over k ^ 2 
gives 

(1 - x)(-$(x,y) - xy) = x^ 2 (x,y) + x(l - x)yV(x,y), 
and the result follows. □ 

In the case of a layered r we get the following counterpart of Theorem 2.3. 
Theorem 3.3. Let r ^ 1 and k = mo > mi > • • • > m r > m r+ i = 0, then 

(1 - xR dm ^{x) xR mr {x))G [ma _ mA {x) = xG^^{x)G^;;2%x) 

/ ' [doj,---,Oj-l,j] v ' [m j ,...,m T ] V /' 

w/iere = — m 3 . 

The proof is similar to that of Theorem 2.3. 

For the case r = 1 one gets the following counterpart of Theorem 2.4, which is 
a generalization of [Kr, Th. 7]. 

Theorem 3.4. For any k > m > 0, 

where z = 1/2^/x. 

Proof. Indeed, by Theorem 3.3 we have 

(1 - xRk-n-^x) - xR m {x))G [kM {x) = xG^l_ 1} (x)G [ ^ ] (x). (3.1) 

Without loss of generality we can assume that 2m ^ k; otherwise it is enough 
to replace [k,m] by [kjin]^ 1 — [k,k — m], since (132)~ 1 = (132). Under this 
restriction, one has gS^^-m (#) = G[ k - m -i](x), and it remains to find g£™ ] ( x). 
Given a decomposition a — (a' , n, a") G 5„(132) as before, it is easy to see that a 
avoids [k, m] and contains [m] exactly once if and only if either a' avoids [k — m—l] 
while a" avoids [k,m] and contains [m] exactly once, or a' contains [m— 1] exactly 
once while a" avoids [m] . We thus get 

n n 

9[r^\ n ) =Y,f[k-m-l](t~ ^g^in-tf+Ysdlrn-l^t- l)/[ m ](n-t), 
t=l t=l 

which on the level of generating functions means 

G^\x) = xF [k _ m _ x] {x)G^\x) + G[ m _i] (x)F[ m ] (x). 

Plugging in the expression for G[ m _i](x) calculated in Example 3.1 and using 
Lemma 4.1 (hi) we get 

r [k,m], n _ Uk-m-l(z) 



a;[/ m (z)C/ m _i(z)C/ fe _ m (z)' 
where z = 1/2^/x. This, together with (3.1) and Lemma 4.1(iv,v) for a = k — m— 1 
and b = m yields the desired result. □ 

One can try to obtain results similar to Theorems 2.5 and 2.6, but expressions 
involved become extremely cumbersome. So we just consider a simplest wedge 
pattern, which is not layered. 
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Example 3.2. Let k > m > p > 0, and let r = {k,m,p} = (p + l,p + 
2, . . . , m, 1, 2, . . . ,p, m + l,m + 2, . . . , k). To find G{k,m. P }{ x ) we use Theorem 3.1 
for r = repeatedly k — m times and get 

G {ktmM {x) = x k - m R 2 k (x) ■ ■ ■ R 2 m+1 {x)G [m ^{x). 

Now Theorem 2.4 yields 

r m = Vx u ™(z) 
<*{k, m , P} W ul{z)Um _ p _ l{z)Up{z) 

with z = 1/2^/x. 

4. Identities involving Chebyshev polynomials 

In this Section we present several identities involving Chebyshev polynomials of 
the second kind used in the two previous sections. We do not supply the proofs, 
since any identity which is rational in z and U p (z) can be proved routinely by a 
computer program. Indeed, it is enough to perform the following steps: 

1) replace z by cos#, and U p (z) by sin(p + 1)0/ sin 8; 

2) since cos6> = (e ie + e~ i9 )/2, and sin 9 = (e ie - e~ ie )/2i, replace cosfl by 
(w + w~ l )/2, and U p (cos6) by { W p +1 - urP- 1 )/^ - u^ 1 ); 

3) the obtained identity is rational in w and can be checked by any computer 
algebra program. 

In the following lemma we assume that R p {x), p ^ 1, is defined by 

R ( x ) - U P- 1{Z) 

with z = 1/2^/x. 

Lemma 4.1. (i) For any s + w — l^t^w^l, 

U s (z)U t {z) - U s+w (z)U t+w {z) = U w -\{z)U a - t+w -\{z). 

(ii) For any s,t ^ and w > 1, 

U s+w (z)U t+w (z) - U s (z)U t (z) = U w - 1 (z)U s+t+w+1 (z). 

(iii) For any p > 1, 

Rp+l{x) = l-xR p (xY 

(iv) For any a,b 1, 

U a +b(z) 



1 - xR a (x)R b (x) 



U a (z)U b (zY 

(v) For any a,b^ 1, 

l-xR a (x)-xR b (x)= ^ U ; + ;^\ 

U a (z)U b {z) 

(vi) For any a > b + 1 ^ 2, 

t/a-6-i(z) 



i? a (x) - i?f,(x) 



V^C/ a (z)C/ 6 (z)' 
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